{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. PP-LCNetV2 Introduction\n",
    "PP-LCNetV2 series model is a lightweight convolution neural network model proposed by PaddleCV team. This series model is improved based on PP-LCNetv1 and focuses more on higher model performance. The accuracy of this model can reach 77.04% on the ImageNet1k dataset. Meanwhile, the inference latency is about 4.32ms. In addition, considering the extensive use of OpenVINO inference framework in the industry, PP-LCNetV2 series model is specially optimized for Intel CPU combined with OpenVINO.\n",
    "\n",
    "The PP-LCNetV2 series model was developed and trained based on PaddleClas. For more information on PP-LCNet series models, see [PaddleClas-PPLCNetV2](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.5/docs/en/models/PP-LCNetV2_en.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Performance\n",
    "\n",
    "The evaluation results of PP-LCNetV2 series models on ImageNet1k dataset are shown as follows:\n",
    "\n",
    "| Model | Params(M) | FLOPs(M) | Top-1 Acc(\\%) | Top-5 Acc(\\%) | Latency(ms) |\n",
    "|:--:|:--:|:--:|:--:|:--:|:--:|\n",
    "| <b>PPLCNetV2_base<b>  | <b>6.6<b> | <b>604<b>  | <b>77.04<b> | <b>93.27<b> | <b>4.32<b> |\n",
    "| <b>PPLCNetV2_base_ssld<b>  | <b>6.6<b> | <b>604<b>  | <b>80.07<b> | <b>94.87<b> | <b>4.32<b> |\n",
    "\n",
    "`_ssld` represents the model after using `SSLD distillation`. For details about `SSLD distillation`, see [SSLD distillation](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.5/docs/en/advanced_tutorials/knowledge_distillation_en.md).\n",
    "\n",
    "Compared with other lightweight networks:\n",
    "\n",
    "| Model | Params(M) | FLOPs(M) | Top-1 Acc(\\%) | Top-5 Acc(\\%) | Latency(ms) |\n",
    "|:--:|:--:|:--:|:--:|:--:|:--:|\n",
    "| MobileNetV3_Large_x1_25 | 7.4 | 714  | 76.4 | 93.00 | 5.19 |\n",
    "| PPLCNetv1_x2_5  | 9 | 906  | 76.60 | 93.00 | 7.25 |\n",
    "| <b>PPLCNetV2_base<b>  | <b>6.6<b> | <b>604<b>  | <b>77.04<b> | <b>93.27<b> | <b>4.32<b> |\n",
    "| <b>PPLCNetV2_base_ssld<b>  | <b>6.6<b> | <b>604<b>  | <b>80.07<b> | <b>94.87<b> | <b>4.32<b> |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Quick Start\n",
    "\n",
    "### 3.1 Inference:\n",
    "* Install the relevant Python packages\n",
    "\n",
    "(Remove the\"!\" when not running on the Jupyter Notebook)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following command to install if CUDA9、CUDA10 or CUDA11 is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install paddlepaddle-gpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following command to install if GPU device is unavailable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install paddlepaddle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install paddleclas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install paddleclas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Quick Start\n",
    "\n",
    "Congratulations! Now that you've successfully installed PaddleClas, you can experience the image classification effects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wget https://gitee.com/paddlepaddle/PaddleClas/raw/release/2.5/docs/images/inference_deployment/whl_demo.jpg\n",
    "!paddleclas --model_name=PPLCNetV2_base  --infer_imgs=\"./whl_demo.jpg\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above command results are as follows:\n",
    "\n",
    "class_ids: [8, 7, 86, 82, 83], scores: [0.8859, 0.07156, 0.00588, 0.00047, 0.00034], label_names: ['hen', 'cock', 'partridge', 'ruffed grouse, partridge, Bonasa umbellus', 'prairie chicken, prairie grouse, prairie fowl'], filename: docs/images/inference_deployment/whl_demo.jpg\n",
    "Predict complete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Training\n",
    "* PP-LCNetV2 series models are implemented based on PaddleClas. For details of model training, please refer to[Training, Evaluation and Inference](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.5/docs/en/models/PP-LCNetV2_en.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Overall Structure\n",
    "\n",
    "The overall structure of the network is shown in the figure below:\n",
    "\n",
    "<div align=\"center\">\n",
    "<img src=\"https://user-images.githubusercontent.com/12560511/200776415-19f7f09d-f8e9-4b64-bbcd-72ac418c23cf.png\"  width = \"80%\"  />\n",
    "</div>\n",
    "\n",
    "### 4.2 Model Details\n",
    "\n",
    "PP-LCNetV2 model is optimized on the basis of PP-LCNetV1. It mainly uses the reparameterization strategy to combine the depthwise convolutions with different kernel sizes, and optimize the point convolution and Shortcut, etc.\n",
    "\n",
    "#### 4.2.1  Re-parameterization\n",
    "\n",
    "The size of the convolution kernel affects the size of the model's receptive field, which affects the model's ability to capture more global or local features. In order to help the model build different scales features, we use 5\\*5, 3\\*3 and 1\\*1 size convolution kernel. The details is shown in the figure below.\n",
    "\n",
    "<div align=\"center\">\n",
    "<img src=\"https://user-images.githubusercontent.com/12560511/201664156-be8f5dc6-5345-436a-962e-079139f378d5.png\"  width = \"80%\"  />\n",
    "</div>\n",
    "\n",
    "#### 4.2.2 PW Conv\n",
    "\n",
    "The depthwise separable convolution usually consists of a layer of DW convolution and a layer of PW convolution to replace the standard convolution. In order to make the depth separable convolution have a stronger fitting ability, we try to use two layers of PW convolution. Meanwhile, in order to control the efficiency of the model, the two-layer PW convolution is set as: The first squeeze the feature map on the channel dimension, and the second expand to restore the feature map channel, as shown in the figure below. Experiments show that this strategy can significantly improve model performance. Meanwhile, in order to balance the impact on model efficiency, PPLCNetV2 only uses this strategy in Stage4.\n",
    "\n",
    "<div align=\"center\">\n",
    "<img src=\"https://user-images.githubusercontent.com/12560511/201664376-e3cdd4ac-0767-4773-a6cc-b6885796bc1e.png\"  width = \"80%\"  />\n",
    "</div>\n",
    "\n",
    "#### 4.2.3 Shortcut\n",
    "\n",
    "The residual structure has been widely used by many models since it was proposed, but in lightweight convolutional neural networks, the elementwise addition operation brought by the residual structure will affect the speed of the model. We experimented on the influence of shortcut on the model at different stage. Finally, we only used Shortcut in the last block, as shown in the figure below.\n",
    "\n",
    "<div align=\"center\">\n",
    "<img src=\"https://user-images.githubusercontent.com/12560511/201664472-42139a13-28a1-43b9-92d7-d355f10ed56d.png\"  width = \"80%\"  />\n",
    "</div>\n",
    "\n",
    "#### 4.2.4 Activation Function\n",
    "\n",
    "In the current lightweight convolutional neural networks, ReLU and Hard-Swish activation functions are most commonly used. Although Hard-Swish is usually better in terms of model performance, we found that some inference platforms have unsatisfactory efficiency optimization for Hard-Swish activation functions. Therefore, in order to give consideration to universality, PP-LCNetV2 uses the ReLU activation function by default, and our tests show that the ReLU activation function has little impact on the performance of larger models.\n",
    "\n",
    "\n",
    "#### 4.2.5 SE\n",
    "\n",
    "Although SE module can significantly improve model performance, its impact on model speed should not be ignored. In PP-LCNetV1, we found that the use of SE module in the back of the model can obtain the maximum benefit. In the optimization process of PP-LCNetV2, we did further experiments on the position of SE module by Stage, and found that better balance could be achieved by using it in Stage4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Notes\n",
    "The inference speed of PP-LCNet series model were tested at Intel Xeon Gold 6271C CPU with PaddlePaddle inference framework, the batch size was 1 and the number of threads was 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Relevant Papers and Citations"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
